Repository navigation
feat(load): gzip large bulk batches and pass them through to Salesforce - #2192
Conversation
Bulk load batches (up to ~10 MB of CSV) upload from the browser through the API server, and upload bandwidth is often the bottleneck. Web batches over 64 KB are now gzipped, and the server forwards the compressed bytes to the Bulk API, which decodes them itself. The server never decompresses the body, so the inflate: false hardening on every other parser stays in place. Desktop, extension and canvas call Salesforce in-process and are unchanged.
![jetstream-bot-agent[bot]](https://avatars.githubusercontent.com/in/4933621?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟢 Ready to approve
No blocking or high-confidence findings: the gzip bytes are never decoded by this server, the inflate: false hardening is intact everywhere else, and the passthrough ordering is covered by a spec that mirrors main.ts.
Reviewed: c04ff4b — 9 files, +227/-10. Focus: request-body handling and body-parser interaction (gzipBodyPassthrough), the Content-Encoding forward to Salesforce (ApiBulk.addBatchToJob), and the client gzip threshold / adapter fallback (getBulkBatchRequestBody).
Checked and clean: passthrough is mounted only on POST /api/bulk/:jobId and before the global raw() parser, which skips it via body-parser's onFinished.isFinished(req) check (body-parser 2.3.0); non-gzip encodings (br, deflate, gzip, x) still reach the global parser and get a 415; over-limit bodies are drained and answered with a 413; non-gzip bytes are rejected with a 400 before anything is forwarded to Salesforce; buffering is capped at the same 30 MB as the existing raw() parsers, so pre-auth exposure is unchanged; desktop, extension and canvas skip compression via AxiosAdapterConfig.adapter; makeRequest in the e2e helper accepts a binary body and custom headers. The five unit tests pass locally, and a manual web load of a ~25 MB, 4-batch CSV completed in Salesforce with every batch gzipped.
🤖 Final-gate review via Claude Code /pr-gatekeeper
Follow-up to #2191.
Bulk load batches (up to ~10 MB of CSV each) upload from the browser through the API server, and on slow or asymmetric connections the upload is the bottleneck. On web, batches over 64 KB are now gzipped in the browser (
CompressionStream). The server forwards those bytes to Salesforce untouched, and the Bulk API decodes them itself. CSV usually shrinks 5–10× with gzip.The server never decompresses anything.
gzipBodyPassthroughis mounted only onPOST /api/bulk/:jobId, before the global parsers. It handles requests that aretext/csvwithContent-Encoding: gzip: it reads the raw bytes up to the existing 30 MB limit, checks the gzip magic bytes, and setsreq.bodyto the Buffer. The global parsers then skip that request, because body-parser checksisFinished(req)first.inflate: falsefrom fix: reject compressed request bodies in every body parser #2191, so any other compressed body still gets a 415.Content-Encoding: gzipthroughApiBulk.addBatchToJob(new optionalcontentEncodingparam).Desktop / extension / canvas: no change. They set a custom axios adapter and call Salesforce in-process, so the client only compresses when
AxiosAdapterConfig.adapteris unset. Their in-process routers read non-JSON bodies as text, so sending them gzip would corrupt the data.Batch size limits are unchanged: the client still splits batches at ~9.5M characters before compressing, and Salesforce enforces its 10 MB limit on the decompressed batch.
Verified
/job/{id}/batchendpoint directly on the dev org withContent-Encoding: gzip: 201, and the rows were inserted (cleaned up afterwards)gzip-body-passthrough.middleware.spec.ts: gzip passes through byte-for-byte, plain CSV still goes through the regular parser, non-gzip bytes get 400, over-limit gets 413,brstill gets 415Bulk Job - Add gzipped Batch: the batch reachesCompletedwith 1 processed, 0 failed (pnpm e2e:local bulk.api.spec.ts; the other bulk API tests pass too)nx affected -t typecheckand unit tests for api, salesforce-api, shared-utils, shared-data, shared-constants, load-records and load-records-multi-object passNot verified: a manual web Load Records run with a batch over 64 KB. The E2E test calls the API directly, so it doesn't exercise the browser-side
CompressionStreampath.